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[57] ABSTRACT 

A novel class of information-processing systems called 
a cellular neural network is discussed. Like a neural 
network, it is a large-scale nonlinear analog circuit 
which processes signals in real time. Like cellular au- 
tomata, it is made of a massive aggregate of regularly 
spaced circuit clones, called cells, which communicate 
with each other directly only through its nearest neigh- 
bors. Each cell is made of a linear capacitor, a nonlinear 
voltage-controlled current source, and a few resistive 
linear circuit elements. Cellular neural networks share 
the best features of both worlds; its continuous time 
feature allows real-time signal processing found within 
the digital domain and its local interconnection feature 
makes it tailor made for VLSI implementation. Cellular 
neural networks are uniquely suited for high-speed par- 
allel signal processing. 
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the cells. The operations of analog systems are based on 
CELLULAR NEURAL NETWORK KirchhofTs Current Law (KCL) and KirchhofTs Volt- 

age Law (KVL). Analog systems have no timing clock 
This invention was made with Government support and work asynchronously. Cellular automata machines 
under Grant No. MIP-86- 14000 and Contract No. S are digital systems and are fundamentally different from 
N00014-86-K0351, awarded by NSF and ONR respec- analog systems, 
tively. The Government has certain rights in this inven- 
tion (3) Interactive Rules 



FIELD OF THE INVENTION 



Cellular neural networks have feedback and feed-for- 
10 ward operations. In feedback operations, there is a self- 
The present invention relates to a circuit architecture feedback function. The connections among cells of the 
for image processing and pattern recognition, namely a network are uniform and local. This means that a cellu- 
cellular neural network (CNN). lar neural network can be characterized by templates of 

BACKGROUND OF THE INVENTION t < its feedback md feed-forward operators. Hopfield's 

15 neural networks have only feedback operations, no 
A CNN is a large scale network of an aggregate of feed-forward or self-feedback operations. The connec- 
regularly spaced circuit clones, called cells, which com- tions neurons are generally global. There is no 
municate with each other non-linearly only through one template concept for Hopfield's neural networks. Also, 
or more layers of the nearest neighbors in real time. cellular automata machines have only feedback opera- 
Conventional digital computation methods have run 20 tioQS , no feed-forward or self-feedback operations, 
into a serious speed bottleneck due to their serial nature. it can be seen that cellular neural networks are 
To overcome this problem, a new computation model. M a ^^Mon of Hopfield's neural networks and 
called "neural networks," has been proposed, which is ^h^. All three of these architec- 
based on some aspects of neurobiology and adapted to tQres m modeb of ^ ^ physical system . They are 
integrated circuits. The key features of neural networks 25 ^ m ^ on ^ to 
are asynchronous parallel processing, contmuous-time model thc ^ physical systcm from different aspects, 
oynamics, and global mteracdon of network elemems. They appear to have similarities because they moddthe 
Some encouraging, if not impressive, applications of H ■ i _* . . • - 
, t °~r° , * J . i ■, same physical system. One cannot derive any one of 
neural networks have been proposed for various fields, \~ f L p . !T V\ " ' 7 

. ^. . . , ,. _ these three architectures from the other two. 
such as. optimization, linear and nonlinear program- 30 

ming, associative memory, pattern recognition and SUMMARY OF THE INVENTION 

computer vision. T _ . . . . 

A new circuit architecture, called a cellular neural In "ontoce with the present invention a new ar- 

network, possesses some of the key features of neural ^ architecture, called a cellular neural network^ 

networks and has important potential applications in 35 wmch <*° P"*™ Processing in real 

such areas as image processing and pattern recognition. 18 *«lowd. In view of the nearest neighbor mter- 

The structure of cellular neural networks is similar to actlve Property of cellular neural networks, they are 

that found in cellular automata; namely, any cell in a much more "nenaMe to VLSI implementation than 

cellular neural network is connected only to its neigh- 8 eneraI neural networks. In spite of the local nature 

box cells. The adjacent cells can interact directly with 40 ofthc nearest neighbor interconnections, cellular neural 

each other. Cells not directly connected together may networks are nevertheless imbued with some global 

afreet each other indirectly because of the propagation properties because of the propagation effects of the 

effects of the continuous-time dynamics of cellular neu- Iocal interactions during the transient regime. This tran- 

ral networks. sient regime can be exploited to show how the ability to 

A cellular neural network differs from Hopfield's 45 analyze the "local" dynamics (via the dynamic route 

neural networks and a cellular automata machine in the approach) will allow the system trajectories to be 

following ways: steered into a configuration of stable equilibria corre- 
sponding to some global pattern that is sought to be 

(I) Mathematical Model recognized. Indeed, the ability to control the local cir- 

Cellular neural networks can be characterized by a 50 c™* dynamics stands out as one of most desirable fea- 

set of difference (space) and differential (time) equa- tures of cellular neural networks. Furthermore, cellular 

tions, which are much like partial differential equations. neural networks have practical dynamic ranges, 

Hopfield's neural networks can be characterized by a whereas, general neural networks often suffer from 

set of ordinary differential equations, which include no severe dynamic range restrictions in the circuit imple- 

space variables. The ordinary differentia] equation 55 mentation state. 

(ODE) and partial differential equation _(FDE) are BRIEF DESCRIPTION OF THE DRAWINGS 
clearly different fields in mathematics. The cellular. 

automata machine can be characterized by two-dimen- FIG. 1 is a two-dimensional simplified block sche- 

sional logic functions. Although it includes the space matic diagram of the present invention showing a CNN 

variables, the logic functions use only binary values for 60 having the interconnections between the cells being 

their input and output signals. Logic function analysis is only to the nearest neighbors. 

also very different from the analysis of differential equa- FIG. 2 is a simplified block diagram showing the 

tions (which use real values as variables). interconnections of a CNN between a single cell and 

„ m , , , each of its nearest neighbors, and self-feedback with 

(2) System Model 65 itself 

Cellular neural networks and Hopfield's neural net- FIG. 3 is a highly simplified representation of a three- 
works are both analog systems. The interactions of cells dimensional configuration of a CNN of the present 
in an analog system depend on the connections among invention. 
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FIG. 4 is an illustration of the relationship of a cell which that link is connected. In this example there is 

and its concentric layers of neighbor cells. only one layer of neighbor interconnects between the 

FIG. 5A is a schematic representation of an equiva- cells with cells 30-44 substantially forming a circle 

lent circuit of a single cell, C,> of a CNN. around cell 12. If there were a second layer of neighbor 

FIG. 5B is a schematic representation of another 5 interconnects, cell 12 would similarly also be intercon- 

equivalent circuit of a single cell, C$, of a CNN. nected with cells 46-58. This layer of neighbor cells 

FIG. 5C is a simplified schematic representation of forms only a partial circle due to the limited size of the 

the equivalent circuit of FIG. 5B. CNN in this example. 

FIGS. 6A-6B are graphical representationsof exam- FIG. 2 shows the extent of the interconnects 80-110 

pies of non-linear characteristics for current source, I. 10 to and from a single cell 60 in a 3x3 CNN array with a 

of FIG. 5. single layer of neighbor interconnects to and from cells 

FIGS. 7A-7D show the initial states, the final states, go, 84, 88, 92, 96, 100, 104, and 108 are 

the output signals at steady state, and the tnmsient volt- fee d-forward links, and links 82, 86, 90, 94, 98, 102, 106 

age waveforms of cell C(2,2) of a 4x4 CNN. ^ n0 m fecdDack Additionally, a self-feedback 

FIGS. 8A-8B are schematic representations of an 15 loopTg from and to cell 60 is shown. To show all of the 

equivalent circuit and a circuit diagram of an opamp mtercormects between each of the cells in this figure, 

miplementanon of a simplified cell circuit of a CNR ^ same leve] ofdetai] would ^ve to be added for each 

FIG. 9 is a gray scale value representation of a 4X4 Q ^ ccJ j s 

P^iPfP' , v - . . . . , , . w . no Next, FIG. 3 is a schematic representation of a simple 
.S^"w X di^SS3^ * P threcHiimensianal 4x4x3 CNN. In this figure, L 

m??^?^ h^f t™™,.*™ „f .ho intersection of each of the row, column and plane vec 

FIG. 11 is a block schematic representation 01 the _ ... , - „ — . , * . „.„ 

interconnection between the laye£ of a multi-layer tors locat ^ <** cel1 «iU ; k) where i 

CNN J represents the row number, j represents the column 

FIG. 12 is a block diagram schematic representation 25 numt f> m ^"~ pr ^^ , u 

of a CNN pattern recognition system. Refemn S to " a V' f ^^^f 

FIG. 13 is a schematic block representation of a 4X4 ~ representauon of a mmti-level CNN having levels 

CNN with the input and output signals multiplexed into U2 ' U6 J^™?* *gnal * being applied di- 

and out of the CNN. rectly to level 122 and m feed forward path 130, via 

FIG. 14 is a schematic block representation of a 4x4 30 delavs 136 and 138, to levels 124 and 126, respectively. 

CNN for inputting signals into the CNN optically. addition, the output signal from level 122 is applied 

to level 124 via delay 132, and the output signal from 

DESCRIPTION OF THE PREFERRED level 124 is applied to level 126 via delay 134. Finally, 

EMBODIMENT there is a feedback path 128 for feeding back signals 

A cellular neural network (CNN) is a multi-dimen- 35 from level 126 to level 124, and from level 124 to level 
sional lattice array of a plurality of identical cells. In a 122. In this implementation the value of delay, 8, for 
two-dimensional planar array of cells, the array may be each of delays 132-138, is substantially equal, 
either square or rectangular. For the n-dimensional In FIG. 4 there is shown a 7x7 matrix of cells with 
case, the array in any direction can have one or more cell 114 at the center of that matrix. Also shown in this 
cells, Le., there does not have to be the same number of 40 figure are three concentric layers of nearest neighbor 
cells in each of the n-dimensions of the array. With cells of cell 114. The innermost layer 116, the next near- 
respect to any single cell, the cells closest to it are its est layer 118 and the outermost layer 120. If the cell 
neighbor cells. Additionally, the neighbor cells of each around which the layers of neighbors are being defined 
cell expand concentrically from that single cell as layers is not centered in the matrix, or if the matrix is rectangu- 
of neighbors. Each cell in the array interacts non-lin- 45 lar, some of the layers will not totally surround the cell 
early and in continuous time with a prescribed number of interest with respect to the layers, 
of layers of its neighbor cells within the lattice. Adja- A typical example of a cell Cfij) of a CNN is shown 
cent cells can interact directly with each other, and cells in FIG. 5A, where the suffixes u, x, and y denote the 
not directly connected together may affect each other input, state, and output, respectively. The node voltage 
indirectly because of the propagation effects of the 50 Vxy of CGJ) is calkd the state of the cell, and its initial 
CNN. A CNN is capable of high-speed parallel signal condition is assumed to have a magnitude less than or 
processing. equal to 1. The node voltage V^is called the input of 

The basic unit of a CNN is the cell. Each cell contains c&j) and is assumed to be a constant with magnitude 

linear and nonlinear circuit elements, which may in- less than or equal to 1. The node voltage Vy$is called 

elude linear capacitors, linear inductors, linear resistors, 55 the output 
linear and nonlinear controlled sources, and indepen- 

d ReSoHG. 1 thereisshown a4x4tw<*»ime»- ^ = - 

sional cellular neural network 10 by way of illustration. . . , _ . n 

The cells C(ij) are shown here in a square array of four 60 l*fr »U 
rows and four columns with T representing the row 

number and "j" representing the column number. In This is an example of a cell circuit wherein C is a 

FIG. 1 there are sixteen cells each shown schematically linear capacitor, R, and R,are linear resistors; Ej/is an 

as being interconnected only to its nearest neighbors, independent voltage source; Ao&fckJ) and Ix/yW) 

i.e., the first layer of neighbor cells. For example, cell 12 65 are linear voltage controlled current sources with the 

{C(2,2)} is shown interconnected with cells 30-44 by characteristics Ij^io;k4)=A(ijW)Vj*/ and J». 

links 14-28, respectively. Each of links 14-28 shown «(iJ;k,l)=B(ijW)V Ii fc/ for all C(ij)€N(iJ); I,«=(l/Rp) 

here are bi-directional links between the two cells to f(Vxij) is a piecewise»linear voltage controlled current 
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source with its characteristic f(.) as shown in FIGS. 6A Constraint conditions: 
and 6B; E/, is an independent voltage, source. 

Observe from FIG. 5A that each cell C(ij) contains |v*f/0)(:s i, lgigM; lSjSN (id) 
one independent voltage source E#» one independent 

current source I, one linear capacitor C, two linear 5 iv«ff|ai. iSiSM; 1SJSN. <ie) 
resistors R x and K y and at most 2 m linear voltage con- 

trolled current sources which are coupled to its neigh- Parameter assumptions: 

to cells via the controlling input voltage Vm and the A ^ mA<ktiA ia4 k s M . m is* (if) 

feedback from the output voltage Vyju of each neighbor ' 

cell C(k,l), where m is equal to the number of neighbor 10 c>a R ^ >0 (lg) 

cells. In particular, Ix/ijW) and WijW) are linear 

voltage controUed current sources with the characteris- Remarks: 

tics I^yW)=A0JW)V y w and I aty (ij;k t l)=B(ij;U. (a) ^ ^ ^ of a neural network have 

)V« W forall (W)€N/ij). Tlie^ the same dxcalt struc tures and element values. The 

in each cell is a piecewise-lmear voltage controlled 15 inner cell is the cell which has <2r+ 1) 2 neighbor cells, 

current source I^=(l/R,)f^^ (Q whcrc r h th e number of layers of neighbor cells which 

of which the characteristics in FIGS. 6A and 6B are m mtei conncct*i. All other cells are called boundary 

two examples. Note, the self-feedback condition is ^ A nctwork ^ completely charac- 

^ 1 • . it j ™ terized by the set of all nonlinear differential equations 

All of toe bnear and piecewise-W controlled » ^ ^ ^ fa ^ 

sources used in our cellular neural networks can be } ^ ^ rf network ^ ^ ^ 

easdy realized usmg operational amplifiers (op amps). A ^ (Sometimes we will choose E ff =0 if 

simple example of an op amp ni.plementat.on of a cell * 0 
circuit is given below. W.toout loss of generality, the 

cell circuit architecture in FIG. 5A will be used 25 rr> ^7 , n ? mc ™ Jf * ww uwca "* tt *~ . .{ 

throughout the specification. Since all cells have the same datum node, and since all 

Referring nexTtoFIG. 5B there is shown a current circuit elements are voltage controlled, cellular neural 
equivalent circuit of a CNN cell C> This circuit has the networks are ideally suited for nodal analysis. More- 
same circuit equations as those discussed above in rela- ov ^ *»cethe mterconnections are local, the associated 
tion to the voltage equivalent circuit of FIG. 5A. To node e ^ uatlon 1S extremely sparse for large cir- 
illustrate the feedback, feed-forward and self feedback cm ^ _ „ , 
loops of the typical CNN cell in FIG. 5C, the voltage ( c > The dynamics of a cellular neural network have 
controlled current sources I^and Uare shown having *>th output feedback and input control mechanisms, 
been replaced by MOVCCS(A) and MOVCCS(B), ^ 0Ut P ut feedback effect depends on the interactive 
respectively, where MOVCCS represents Multiple 35 parameter A(ij;k,l) and the input control effect depends 
Output Voltage Controlled Current Sources. The out- on B(ij;k,l). Consequently it is sometimes instructive to 
put currents from MOVCCS(A) are the feedback cur- refer to A(ij;k4) as a feedback operator and B(ij;k,l) as 
rents with the exception of the current where i=k and a control operator. The assumptions in (If) are reason- 
j=l, that is the self-feedback current. The output cur- able because of the symmetry property of the neighbor- 
rents from MOVCCS(B) are the feed-forward currents. 40 hood system. Where i=j, A(ij;k,l) is a self-feedback 

In FIG. 6A the representative characteristic for cur- operator, 

rent source, I, in FIGS. 5A-C is shown as being a one- (d) The values of the circuit elements can be chosen 
step sigmoidal function. Similarly, in FIG. 6B the char- conveniently in practice. R* and K y determine the 

acteristic represented is that of a multi-step sigmoidal power dissipated in the circuits and are usually chosen 

function. 45 to be between Ikft and 1MO. CR X is the time constant 

Applying Kirchhofs current law and KirchhoPs of the dynamics of me circuit and is usually chosen to be 
voltage law, the circuit equations of a cell are easily 10- 8 — 10- 5 sec. 

derived as follows: _ . _ _ _ „ . _ . * ~ _ _ 

State equation- A * Dynamic Range of Cellular Neural Networks 

50 Before discussing the design of a physical cellular 
tit^/t) 1 (la) neural network, it is necessary to know its dynamic 



c X " ~"rT Vx ^' } + CXJt/)^^)^^ 1 ^ 0 + range in order to guarantee that it will satisfy the as- 

sumptions in the dynamical equations stipulated above. 

I BiijMvuk] + I The following theorem is the foundation of the CNN 

GftfcWiW 55 dcsign 

1S/S*1S7*M Theoreml 

Output equation: AD states in a cellular neural network are 

bounded for all time t>0 and the bound V,^ can be 



60 



W> = 4" c|Vjc * f<0 + 11 ~ ,Fj ^° ~ ° b> neural network: 

1 £ 1 3 M: 1 S J S JV. 

max[l(\A(iJ:lc r r)\ + \IKlJ;k r !)\)l iSiSM. 
Input equation: « lSjSN CfrDiNAi) TO 

Vuij^Eij, 1 *i£M; \*j£N. (lc) For any cellular neural network, the parameters, R x , C, 

I, A(iJ;k,l) and B(i j;k,l) are finite constants, therefore 



computed by, the following formula for any cellular 
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the bounds on the states of the cells, V mautt are finite and 
can be computed via formula (2). 

In actual circuit design, it is convenient to choose the 
scale of the circuit parameters such that Rjc|I|=l, 
Rx| A(io;k,l)| =; 1 and R*|BGj;k,l)| si, for all i, j ( k, and 
L Hence, the upper bound on the dynamic range of the 
cellular neural networks can easily be estimated. For 
example, if a neighborhood of the cellular neural net- 
work is 3x3, V^sM V is possible, which is within 
the typical power supply voltage range of IC circuits. 

B. Stability of Cellular Neural Networks 

One application of cellular neural networks is in 
image processing. The basic function of a cellular neural 
network for image processing is to map, or transform an 
input image into a corresponding output image. Here, 
the output images are restricted to binary images with 
— 1 and +1 as the pixel values. The input images can 
have multiple gray levels, however, provided that their 
corresponding voltages satisfy (le). This means that an 
image processing cellular neural network must always 
converge to a constant steady state following any tran- 
sient regime which has been initialized and/or driven by 
a given input image. 

One of the most effective techniques for analyzing 
the convergence properties of dynamic nonlinear cir- 
cuits is Lyapunov's method. Hence, it is necessary to 
define a Lyapunov function for cellular neural net- 
works. 

Such a Lyapunov function, E(t), for a cellular neural 
network is defined by the scalar function: 



8 

Theorem 3 



The scalar function E(t) defined in (3) is a monotone- 
decreasing function, that is, 



5 0. 



(5) 



10 



According to our definition of cellular neural networks, 
we have: 



h4<. a * L -[«j«.'" M -- 

15 



20 



Substituting the cell circuit equation (1) into (6), and 
recalling C>0 in assumption (Ig), we obtain: 



25 



* i*fl<i c L * J ac 



(7*) 



30 



Remarks: 

For future analysis (e.g., corollary to Theorem 4), it is 
convenient to rewrite (7a) as follows: 



(3) 



35 



Remarks: 

(a) Observe that the above Lyapunov function, E(tX 40 
is only a function of the input, V„, and output, Vy, volt- 
ages of the circuit. Although it does not possess the 
complete information contained in the state variables 
Vxijf we can nevertheless derive the steady-state proper- 
ties of the state variables from the properties of £(t). 45 

(b) The Lyapunov function, E(t), defined above, can 
be interpreted as the "generalized energy** of a cellular 
neural network, although its exact physical meaning is 
not very clear. As the following theorems will show, 
E(t) always converges to a local minimum, where the 50 
cellular neural network produces the desired output 

Theorem 2 

The function £(t) defined in (3) is bounded by 




(7b) 



s a 



From Theorems 2 and 3, we can easily prove the fol- 
lowing important result: 

Theorem 4 

For any given input V« and any initial state V x of a 
cellular neural network, we have 



hm £(/) = 



55 



<*0 



mu|£(i)| 5 E m 
t 



(8b) 



where 



Emax — 



60 



(4b) 



w (Tfer + ii) w 



for an MXN cellular neural network. 



From Theorems 2 and 3, £(t) is a bounded monotone 
decreasing function of time t Hence £(t) converges to 
a limit and its derivative converges to 0. 

Corollary 

After the transient of a cellular neural network has 
decayed to zero, a constant dc output is always ob- 
tained. In other words, 
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-continued 

(8c) (11b) 
lira vjfft) - constant, \£ i£M;\*j£ N Urn v^//) = rtl, ISiSM'IS/S/V 

5 

or 

Remarks: 

(Bd) 

dvyjft) (a) The above theorem is significant for cellular neu- 

t™ m dt ° °' 1 3 / 3 ^ 1 S; 5 * ral networks because it implies that the circuit will not 

10 oscillate or become chaotic. 
Next, investigate the steady-state behavior of cellular <*>> ^corem 5 guarantees that cellular neural net- 
neural networks. It follows from Theorem 3 that under wo ^ haye btiiary.valuc outputs This property is cm- 
the condition (dE(t)/dt)=0, there are three possible cial for solving classification problems m image process- 

cases for the state of a cell as t tends to infinity i« ^PP" 03 " 0 ^- , . . . . 

13 (c) It can be easily shown by the same technique that 

without the constraint of (10) both case (1) and case (2) 

(1) ^l^lL = oand \vxiAt)\ < i ^ can co-e^is* but case (3) cannot. This implies that re- 

* * mark (a) is true even without the condition (10). 

^ (d) Since A(i j;i j) corresponds to a feedback from the 

(2) ffi = Oand |v x ///)| £ i 20 output of cell C(ij) into its input, condition (10) stipu- 

lates a minimum amount of positive feedback in order to 
dvxi/t) (9c) guarantee that the steady-state output of each cell is 

(3) dt * Oand 11*0(01 * i either +1 or -1. Note that this condition is always 

violated in a Hopfield neural network since its diagonal 

because of the characteristic of the piece wise-linear 25 coupling coefficients are all assumed to be zero. To 

output function (lb). guarantee a similar ± 1 binary output in the Hopfield 

This will be clear upon consideration of FIG. 6A. model, it is necessary to choose an infinite slope in the 

When |Vx#(t)| <1 we have V^t)=V^(t) and there- linear region of the nonlinear function f(.) in FIG. 6A. 

fore (dV y (/(t)/dt)=(dVj C ^(t)/dt). From Theorem 4 and In contrast, the corresponding slope in a cellular neural 

its corollary, case (1) follows. But for | V^t) | > 1, since 30 network is always chosen to be equal to one. 

y^t)]V^t), when Vjtft)«±l is a constant, we do not Q Sim , e Nemal Network 
have the precise waveform of V x i/J0 In the case where 

V„y(t)= constant, we have case (2). Otherwise, case (3) A verv simple example to illustrate how the cellular 

applies and V^/t) may be a periodic or an aperiodic, but neural network described above works is presented 

bounded, function of time, in view of Theorem 1. 35 here * T ^ s example will also help to provide a better 

Returning now to the basic cell circuit (FIGS. understanding of the theorems of the preceding sec- 

5A-5C) of the cellular neural networks, it can be seen tarns. 

that since the output signal of the time varying indepen- Thc cellular neural network 10 for this example is the 

dent current source is a function of only the output ^ sa ^ e f shown m nG *• that network size 

voltages, V,**), and the input voltages, V M of the 40 * 4x4. The circuit element parameters of the cell C(ij) 

neighborhood of the cell, it follows from the results of ^chosen as follows. . 

Theorem 4 that all of the steady-state output signals of ^ZJZ^V^^ " = ^ 

our cellular neural network are constants. Hence, after neignnorncoa cells;, let 
the initial transients, the assumption that the time vary- 45 

ing independent current source output signal is constant ^jjj ~ ^q- 3 !)- 1 

is valid for the study of the steady-state behavior of 4^ _ \j + \) - 0 % 

cellular neural networks. Summarizing the above obser- ^(4/rv - 1) - )0- 3 Q- >; 

vation: ^CAav) = 2.0 x lo- 3 n-> : 

A{ijij+ i)= lo-^n- 1 

Theorems 50 -iGtfrf+V-D-ft t 

If the circuit parameters satisfy + ljf + l) = 0 

c = \o- 9 F; R x = 10*0; / = 0 
J(UJM) = 0. for GtM)«Wv). 



33 Since B(y ; k4)=0 > the 3x3 coefficients A(ij;k,l) 
alone determine the transient behaviors of the cellular 
then each cell of the cellular neural network must settle neural network. Those coefficients win often be speci- 
al a stable equilibrium point after the transient has de- fied m ^ f orm D f a sq^c matrix, henceforth called the 
cayed to zero. Moreover, the magnitude of all stable cloning template, which specifies the dynamic rule of 
equilibrium points is greater than 1. In other words, we 60 the cellular neural network. 

have the following properties: The dynamic equations of the cellular neural network 



corresponding to the above parameters are given by 



(Ha) 



Km |»x0to| > 1. IS/SJIi-lSySAf M d^Jji) (12a) 

dt ° 



and 
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-continued 

and 

tyflQ " "(MO + J l - l*0(J> - 1 IX (Hb) 
for 1 S / S 4; 12)2 4. 



05) 

= Oand (v-^j > 1 



under the assumption Vykl= ± 1 for all neighbor cells 
It is convenient to recast the right-hand side of (12a) C0c,l)eN,(ij). 
into the symbolic form Equation (15) holds for any assumed combination of 

10 Vyki=±\ t and, therefore, may not represent an actual 

[_ (13) component of an equilibrium state of the overall circuit. 

Oil For the current example, equivalent! y, the stable cell 

1 2 1 I equilibrium states of a cell circuit C(i J) are the solutions 

o i o JV/') V xi j of the dc cell circuit equations obtained by replac- 

15 ing all capacitors by open circuits; namely. 



with the help of the two-dimensional convolution oper- 
ator * defined below; Ae conditions 

For any cloning template, T, which defines the dy- 20 
namic rule of the cell circuit, we define the convolution 
operator • by ^ 



|Vjcfc/|£l, ISk, 134 (16b) 



— . N _ , . , _ . Substituting condition (16c) into the dc equation 

where T(m,n) denotes the entry in the mth row and nth (16d) ^ ^ the , * defined by 

column of the cloning template, m= — 1,0,1 and 

n= — 1,0,1, respectively. 

Note that in the above definition A(i j;k,l) is assumed 30 l. * > 0 

to be independent of i and j for this cellular neural net- a 0, x « o 

work. This property is said to be space invariant, which - . * < 

implies that A0j;k,l) can be expressed as A(k— U — j). Y .._ 2sg j^y *+sgnty i/)+«n(K 

Unless stated otherwise, all cellular neural networks are 0+JSP»(*W+ if" * (17) 

assumed to have the space invariant property. This 35 

property allows us to specify the dynamic rules of cellu- Furthermore, since sgn (Vjy)=sgn(V X ij) from (lb), it 

lar neural networks by using cloning templates. follows that 

To study the transient behavior of (12a), apply an 

initial voltage V^O) across the capacitor of each cell Vxv-i^Vx(j)~sgn(Vyi-ij)+sgn{Vy{+ij)+sgn- 

C(ij). Each initial voltage may be assigned any voltage 40 (*j#- i)+*K^+ iX (18) 
between — 1 and +1, as stipulated in (Id). 

The transient behavior of the above cellular neural Observe that the right-hand side of equation (18) can 

network with the initial condition specified in the array assume only five possible values; namely, -4, -2, 0, 2, 

of FIG. 7A has been simulated. The state variables of m & 4 - 11 follows that the corresponding values that can 

the circuit V,, at t=5 u-S are shown in FIG. 7B. The 45 assumed by the state variable V^are -6, -4, (-2 

maximum absolute value of the state variables at t=5 or and 6. 

jtS is equal to 6, approximately. The upper bound V max lt follows from the above analysis that each inner cell 

of Vjr as computed from equation (2) of Theorem 1 is circuit for our present example can have only six possi- 

equal to 7, which is very close to 6, ble stable cell equilibrium states; namely, —6, —4, —2, 

The corresponding output voltages, at t = 5 ftS 50 2, 4, and 6. 

are shown in FIG. 7C. Observe that all output variables The actual stable cell equilibrium state attained by 

assume binary values, either -f 1 or — 1, as predicted by each cell clearly depends on its initial state as well as on 

Theorem 5. (Here the condition A(ij;ij)> l/R x is satis- those of its neighbor cells. Hence a cell may eventually 

fied.) approach any one of its stable equilibrium states even if 

The transient behavior of cell C(2»2) is displayed in 55 its initial state remains unchanged; 

FIG. 7D. The initial value of the state variable is equal A stable system equilibrium point of a cellular neural 

to 1.0, and the value at t=5 uS is equal to 2.02, approxi- network is defined to be the state vector with all its 

mately. The maximum value of Vx22(t) is equal to 3 and components consisting of stable cell equilibrium states, 

occurs at t=0.8 u-S, approximately. Since the state van- It follows from the above definition that a cellular 

able is kept above 1.0 during the entire transient regime, 60 neural network is always at one of its stable system 

the corresponding output remains constant at 1.0, as equilibrium points after the transient has decayed to 

predicted from FIG. 6A. ze~o. From the dynamic system theory point of view, 

Before investigating how many distinct values V^t) the transient of a cellular neural network is simply the 

can assume in the steady state, consider first the follow- ' trajectory starting from some initial state and ending at 

ing: 65 an equilibrium point of the system. Since any stable 

A stable cell equilibrium state, V* x /> of a typical cell system equilibrium point, as defined above, of a cellular 

of a cellular neural network is defined as the state van- neural network is a limit point of a set of trajectories of 

able Vxijof cell C(ij), which satisfies the corresponding differential equations (1), such an 
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attracting limit point has a basin of attraction; namely, 
the union of all trajectories converging to this point. 
Therefore, the state space of a cellular neural network 
can be partitioned into a set of basins centered at the 
stable system equilibrium points. 

Generally speaking, a cellular neural network pro- 
cesses signals by mapping them from one signal space 
into another one. In our example, the cellular neural 
network can be used to map an initial state of a system 
into one of many distinct stable system equilibrium 
points. If we consider the initial state space as 
[-l-ftl-OyKxiV and the output space as {-hi}****, 
then the dynamical map F, can be defined as 



14 



(20) 



10 



Then, for multilayer cellular neural networks, the cell 
dynamic equations can be expressed in the following 
compact vector form: 



(19) 15 



1 £ / £ M; 1 S j £ N 



(21) 



This means that the map F can be used to partition 
continuous signal space into various basins of attraction 
of the stable system equilibrium points via a dynamic 
process. This property can be exploited in the design of 20 
associative memories, error correcting codes, and fault- 
tolerant systems. 

In general, the limit set of a complex nonlinear system 
is very difficult, if not impossible, to determine, either 
analytically or numerically. Although, for piecewise- 25 
linear circuits, it is possible to find all dc solutions by 
using either a brute force algorithm or some more effi- 
cient ones, it is nevertheless very time consuming for 
large systems. For cellular neural networks, in view of 
the nearest neighbor interactive property, all system 30 
equilibrium points can be solved by first determining 
the stable cell equilibrium states, and then using the 
neighbor interactive rules to find the corresponding 
system equilibrium points. 

The dynamic behavior of a cellular neural network 35 
with zero control operators and nonzero feedback oper- 
ators is reminiscent of a two-dimensional cellular au- 
tomaton. Both of them have the parallel signal process- 
ing capability and are based on the nearest neighbor 
interactive dynamic rules. The main difference between 40 
a cellular neural network and a cellular automata ma- 
chine is in their dynamic behaviors: The former is a 
continuous time while the latter is a discrete-time dy- 
namical system. Because the two systems have many 
similarities, cellular automata theory can be used to 45 
study the steady state behavior of cellular neural net- 
works. Another remarkable distinction between them is 
that while the cellular neural networks will always 
settle to stable equilibrium points in the steady state, a 
cellular automata machine is usually imbued with a 
much richer dynamical behavior, such as periodic, cha- 
otic and even more complex phenomena. Cellular neu- 
ral networks have been tamed by choosing a sigmoid 
nonlinearity. If some other nonlinearity were chosen for 
the nonlinear elements, many more complex phenom- 
ena would also occur in cellular neural networks. 



50 



55 



D. Multilayer Cellular Neural Networks 

The single-layer cellular neural network of FIG. 1 
can be generalized to a multilayer cellular neural net- 60 
work. Instead of only one state variable in the single- 
layer case there may be several state variables in each 
cell of a multilayer cellular neural network. The con- 
cept of multilayering emphasizes the interactions of the 
state variables on the same layer. To avoid clutter, it is 65 
convenient to use the convolution operator * in the 
following discussion. Using the convolution operator, 
equation (la) can be rewritten as 



o o 

0 



R - 



0 0 
0 



Q 
0 

0 0 
0 

0 0 



An o o o o 

. 0 0 

. 0 0 

. 0 

A m i . . . A m 



f Bu 0 0 0 0 X 
. 0 0 
. 0 0 
. 0 

2W • • • B m 



(22a) 



(22b) 



Vxff « 











f \ 










Viuij 













(22c) 



(22d) 



and where m denotes the number of the variables in the 
multilayer cell circuit Here, the convolution operator * 
between a matrix and a vector is to be decoded like 
matrix multiplication but with the operator * inserted 
between each entry of the matrix and of the vector. 



06/16/2003, EAST Version: 1.04.0000 



(27) 



5,140,670 

15 16 

Observe that C and R are diagonal matrices, whereas the pixel values. However, the input images can have 

A and B are block triangular matrices. multiple gray levels, provided that their corresponding 

Remarks: voltages satisfy equation (le). This means that the image 

processing cellular neural network must always con- 

(a) For multilayer cellular neural networks, all of the 5 verge to a constant steady state following any transient 
results presented in the previous sections still hold with regime wh ich has been initialized and/or driven by a 
some minor modifications. The stability can be proved g ven mput image, it has also been proven that cellular 
from the bottom layer (layer 1) to the upper ones by neural networks m stable. 

notmgtfie block triangular structures of the A and B A circuit ^ ^ t0 ^ "completely stable" if and only 

"rtCk *aL~ *~ c~,~*i tf ^ AU ;« „ 10 if every trajectory tends to an equilibrium state. Conse- 

(b) Since there are several state variables m a cell ^ ^ k { cannot oscillate or become cha- 
circuit, multiple dynamic rules can be chosen concur- ^ c 

rently for the different state variables. This property °~r' . . A . . . . , , 

makes the network extremely flexible and allows more After the transient has settled down, a cellular neural 
complicated image processing problems to be dealt ,< n**** dways approaches one of its stable equilibrium 

13 points. In other words, 

(c) In addition to using multiple dynamic rules as 

mentioned in (b), different time constants can be chosen (26) 
for the different state variables of the cell circuits. As a ^ = consunt I £ / £ M r ] s j g w 

limiting case, C 9 =0 can be chosen for some state vari- ^ 
able y<pjp thereby obtaining a set of differential and 
algebraic equations. This property yields even more OT 
flexibility in the design of cellular neural networks for 
practical problems. 

The circuit in FIG. 8A is an equivalent circuit of a ^ di - i £ / 3 J* I sy 3 M 

simplified cell circuit of cellular neural networks. It * 

consists of the basic circuit elements, namely, a linear Moreover, if the circuit parameters satisfy 
capacitor C; a linear resistor R x ; a linear voltage con- 
trolled current source Ijq<ij;k,l)=AOj;k,l)V^/; a sub- 
circuit with the piece wise-linear function Vt#=>0.5(| V x „ rt A{Q;ij) > -4— W 
tft)+l|-|V*>(t)-l|). *> 

One possible op amp implementation of the above then 
circuit is shown in FIG. 8B. The voltage controlled 

current source Ixy(vj;k,l) is realized by op amp Ai and (29) 
resistors R1-R3.lt can be shown that lim |vx//o| s u 1 £ t £ M; 1 sy s N 

W = W <23) or cquivalcntiy. 

(30) 

under the condition that ^ Uid v /o = ±1, \£i£M;\mj£N. 

*2 + *3 (24) 

= — 355 — This is significant for cellular neural networks, be- 
cause it implies that the circuit will not oscillate or 

The output resistance of lWijW) is infinite under con- 45 bccomc chaotic - Further, satisfaction of equation (28) 

dition (24). The piece wise-linear function Vyi/Yxfj) is guarantees that the cellular neural network will have 

realized by op amps A* A 3 and resistors R0-R9 with the binary-value output values. This property is crucial for 

constraint that solving classification problems in image processing ap- 
plications. 

Rt + *7 *a + *9 (25) 50 In order t0 Perform signal processing, pattern recog- 

— — = — ^ — + I ^crl nition, or other functions that a CNN can be called 

where V« is the voltage of the power supply. U P° D to . ^°™' a configuration similar to that 

The two circuits in FIGS. SA and 8B are equivalent ^ ov ? m FIG. 12 will be necessary. In the system an 

in the sense that they have the same state and output interfacc svstem 132 » used to apply the image to be 

equations. 55 processed simultaneously onto each CNN 136-x in the 

In the following example, a cellular neural network is ^ nern recognition block 134. The output signals from 

used to solve image processing and pattern recognition ^ Ac CNNs 136-x in block 134 are supplied to the 

problems. Only the steady-state behavior of cellular decision making block 138. 

neural networks is stressed. However, for applications For example, if the circuit of FIG. 12 is to be used to 

in image processing, the transient behavior is equally 60 determine if the image that is being applied to the 

important. In fact, it is the transient behavior which TABLE 1 

makes it possible to extract a variety of features from a ^ rfT - Mathmfltiptl M . . „ . o 

single picture, and to solve various image processing Compmoa ^ 

problems. Cellular Partial 

The basic function of a cellular neural network for 65 Neural Differential 2-D Cellular 

image processing is to map or transform an input image Model Network Equation Automata 

into a corresponding output image. Here, we restrict ^ continuous continuous dbc^ 

our output images to binary images with — 1 and + 1 as space discrete continuous. discrete 



06/16/2003, EAST Version: 1.04.0000 



5,140,670 

17 18 

TABLE I-continued 

r - (3l> 



Comparison of Three Mathematical Models Having 
Spatial Regularities 




Cellular 


Partial 






Neural 


Differential 


2-D Cellular 


Model 


Network 


Equation 


Automata 


state 
value 
dynamics 


real 
nonlinear 


real 
linear 


binary 
number 
nonlinear 



/va Ar « a A MIQ VA») + hi \ 

1 ^ | g M; \ m j I £ N 

10 where the time step h has been suppressed from "nh" 
pattern recognition block 134 is that of a chair with for simplicity; i.c, Vj^n^Vj^nh) and V^/n)- 
certain characteristics, each of CNNs 136-x will have a 

=V^nh). 

circuit element for each pixel in the image. Each pixel of Substituting equation (29b) for V yW (n) in equation 
the image is applied to the corresponding circuit of each 15 (3*)' 
CNN in bank 134. Additionally, each CNN 136-x is 

programed with a different cloning template, wherein r _ < 32 ) 

each cloning template is selected to enable the CNN to + l) - i^n) + -g- -jg- + 

which it is applied to detect one characteristic of the 

chair of interest, e.g., CNN 136-1 has been programed 20 . 

to look for a convex corner, CNN 132-2 has been pro- ctk.t?NJ' ^^MyxkM) + hi ]. 

gramed to look for a concave corner, and CNN 132-n w J 

has been programed to look for a hole. The input image is/s^-i SjSJV.. 

could also be examined by numerous additional CNNs _ . . J J . 

to look for additional features. Here it should be noted 25 Equation (32) can be interpreted as a two^ensional 

that whether one feature, or 100 features, are looked for ? ter for transfonmng an image, re^tedby V^n), 

in an image, one need only add additional CNNs to ^to mother one, represented by V*(n+ 1 l The filter is 

perform the desired functions. The CNN bank requires f^<n) » m equafcon (32) is a nonhn- 

no more time to look for 100 characteristics than h does J^T^^ 

for 1 since CNNs 136-x potions in pan* jr with each 30 ^^e X*SE 

other. In a simple application^ such as the chair recogni- ffl|cr ^ iSmM by the parameters in equation (32). 
tion system the output signal from each CNN can be a Fof ^ mter fa (32) ^ p ^ Vfl] . 

binary signal. ues, V^n+l), of an image are determined directly 

Decision making block 13S could be a state rn^hme from & pixel values , V^n), in the corresponding 
that takes the binary information from the CNNs and, neighborhood. From the practical point of view, this 
from those signals, determines if the image was that of a neighborhood is always chosen to be as small as possi- 
particular type of chauv Decision block 138 might also bl& A ^ ical choice ^3x3 neighborhood. There- 
be another bank of CNNs which processes the signals forE) a onMte p fflter ^ only ^ of th c \qc^ 
from pattern recognition bank 134 to determine if the ^ properties of images. 

type of chair that was imaged onto bank 134 was of a when the global properties of an image are impor- 
particular design represented by the individual cloning t^nt, the above one-step filter can be iterated n times to 
templates used m the CNNs of decision block 138. In extract additional global information from the image, 
order to understand why cellular neural networks can One well known property of the iterative filter is the 
be used for image processing, first approximate the 45 so-called propagation property, which asserts that the 
differential equation (la) by a difference equation. In pixel values of the output image after n iterations can be 
particular, let t=nh, where h is a constant time step, and indirectly affected by a larger neighbor region of the 
approximate the derivative of VxijCX) by its correspond- input image. This property can be observed by substr- 
ing difference form tuting V»/n) in equation (32) iteratively down to t^O), 

50 which coincides with the input image. Since equation 
C r „ , v . v . t « 1 , ^ G9a) (32) contains some nonlinear functions, it is convenient 

extort^ <wwv,w> (33) 

and 

oqm where g n ijk! is a nonlinear function, which depends on 
vyifixh) = OSWvxfoh) + 1 1 - |y„/nA) - i|) (jj), (kj), and n. Note that the larger neighborhood is n 

_ j(? x finh)\ 60 times larger than the local neighborhood and depends 

i s i 2 Af; i s j £ a: on the iteration number n. It is easy to see that when the 

iteration number n is sufficiently large, the larger neigh- 
Lct borhood will eventually cover the entire image. 

//.- 2 muMvu + L (30) Therefore, the propagation property of iterative fil- 

3 CfabtNAtf) " 65 ters makes it possible to extract some global features in 

l £ / ^ M; i ^ y s N. images. Of course, the local properties are still pre- 

served in iterative filters with the closer neighbors hav- 
recasting equation (29a) into the form ing more effect than those farther away. 
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Now, it is not difficult to understand how cellular element parameters of the cell C(i j) are chosen as fol- 

neuraJ networks can be used for image processing. In- lows: 
deed, if h— *0 in equation (29a), the system equation 

(la-g) defining a cellular neural network is recovered. c a IQ _ 9 f;R _ IQ j n / ^ 

To understand the image transform mechanism in cellu- S = * ' = 

lar neural networks, rewrite equation (la) in its equiva- A ^ :i ~ lJ ~ l) 53 A ^ ;i ~ ^ 

lent integral form as follows: = *W - U + 0 = 0 

^(vX/ - i) = ^(«v + I) = io- 3 n-» 

1 3 lAf; 1 3 j 3 AT, 

for a 3x3 neighborhood system. Since the feedback 
Jfti)- 2 ^*0W) (Mb) operators AfijjkJ), as shown above, are independent of 

cxkbtNAij) j 3 the absolute position of the cell, we can simplify the 

expressions of A(iJ;k,l) by coding them as follows: 

mo 

" ™ Am* a ******** (34C) iK-l.-l) = i«-lfl> - ^(-U) = 0 

Ww»f(y) ^(0,0) e 2 x io- 3 

20 -W -I) -^(0,1)- 10- 3 Q 

Equation (34a) represents the image at time t, which A(\;-\) = A(\Xf) = >4(U) = o. . 

depends on the initial image V^O) and the dynamic 

rules of the cellular neural network. Therefore, we can The indices in the above interactive parameters indicate 

use a cellular neural network to obtain a dynamic trans- the relative positions with respect to C(i j). The cellular 
form of an initial image at any time t In the special case 25 neural network has the space invariance property, that 

where t->oo, the state variable V x y tends to a constant is, A(iJ;k,r)=A(i-kj-l) for all i, j, k, and 1. Therefore, as 

and the output Vj^* tends to either + 1 or — 1 as stated in in image processing filters, a cloning template, as shown 

equation (30). in FIG. 10, can be used to describe the feedback opera- 

E. A Simple Image Transform ' tor ? f henceforth called a feedback operator 

30 cloning template. The cloning template is constructed 

Before considering real image processing problems, it & follows: the center entry of the cloning template 

is instructive to look at a very simple example. Al- corresponds to A(0,0); the upper left corner entry of the 

though it is a much sjmphfied image processing prob- cloning template corresponds to A(-l,-l); the lower 

lem, this example will help to understand some of the right-hand corner entry of the cloning template corre- 

dynannc behavior of cellular neural networks and to s ds to A(1 1} ^ M forth . From th/aW ^ 

derive some intuitive ideas on how to design cellular ™ . . , , , . , , . 

«i * 1 . r 7 • ~£z \- following discussions it should be clear that given the 

neural networks for solving a specific practical image . . , f ^ - « vtVT Al _ r 

processing problem. * dc *™* 1 fim f on of a °NN- the recognition of a 

One important problem in image processing is pixel P^ cula r character orshape, the individual [templates 

classification. To illustrate thTconcept, consider a , A for «* of * c ™ can be derived. Further it 

small image, the gray scale representation of that image 40 f" 0 ^ * ^ template of each cell is loaded 

is a 4X4 pixel space shown in FIG. 9. This image is a mto that ^ bv Meeting the circuit element and gain 

4X4 pixel array with each pixel value P/ye[-l,l], for vaIues to Provide the feedback, self-feedback and feed- 

l^i^4 and l^j^4. Assume that the pixel value -1 forward values specified by the -template. Since it is 

corresponds to a white background, the pixel value + 1 extremely convenient and clear to characterize the in- 

corresponds to a black object point value, and the pixel 45 teractions of a cell with its neighbors by means of a 

values between —1 and + 1 correspond to the gray cloning template, the cloning template expression will 

values. The pixel classification problem is to classify be used in the following examples, 
each pixel of the image into two or more classes. The dynamical equations of the cellular neural net- 

From the mathematical point of view, pixel classifies- work corresponding to the above parameters are given 

tion can be considered as a map, F, which maps a con- 50 by 
tinuous vector space into a discrete vector space as 
defined below: 

/gfe^JfeW-^UAG . . . (35) 

55 

where MxN is the number of pixels in an image and 
A,B,C, . . . stand for different classes. For this example, V/0 = asflwo + 1 1 - - 1 \% (37b) 

we wish to assign to each pixel in the array one of the far 1 3 / s 4 ; 1 5/ 3 4> 

two values, — 1 and + 1, based on some classification VT . , „ . . , v ^ m 

rules and the original pixel values. So, F is defined by 60 Note that the control operator BOo W)=0 for all 1, j, k, 

and 1 in this circuit, and A(ij;k4)>l/R x as stipulated in 

fii-utoxn***— {-u}"** (36) the condition of equations (28H30). 

For this example, the initial state of the cellular neural 

Suppose the object is to design a horizontal line de- network is the pixel array in FIG. 9. From the circuit 

tector to filter out the horizontal lines in the input image 65 equation (37), we note that the derivative of the pixel 

in FIG. 9 by using a cellular neural network. In order to values depends on their left and right neighbors. This 

simplify the analysis, a very simple dynamic rule for this particular dynamic rule will therefore enhance the de- 

"horizontal line detector" circuit is chosen. The circuit tection of horizontal lines in the original image. 



and 
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The circuit equations (37 and 37b) are first-order ested only in those cell equilibrium states having a mag- 
nonlinear ordinary differential equations. In system nitude greater than one. 
theory, they are also called a piecewise-linear autono- 
mous system. In general, it is difficult if not impossible Definition 2: Stable Cell Equilibrium States 
to predict the behaviors of complex nonlinear dynami- 5 A cell equilibrium state V* XI y of a ceD circuit C(ij) is 
cal systems. The analysis above shows that the circuit in said to be stable if and only if 
this example will tend to one of its equilibrium points 

after its transient has settled down. |V*^|>1. (40) 

The equilibrium points (steady state) of the system 
can be found by solving the equivalent dc circuit equa- io Observe that since the observable (i.e., stable) output 
tions (replace all capacitors by open circuits): variable of each cell C(iJ) can assume only the 

values + 1 or — 1, h follows that any observable output 
»xt/(') = *yu- i(f) + 2vrfQ + *w+ (38a) solution vector, Vy, of any cellular neural network with 

A(ij;k,l)> l/K x must necessarily be located at a vertex 
^/s^'s^ 1 ' " l?xfl< ' ) " ,|X 08b> 13 of m n-dnnensional hypercube S, where n=MxN 

~ ' denotes the total number of cells. If we let S 0 denote the 

In general, for piecewise linear circuits, all solutions set of all n-dimensional vectors, whose components, 
of the dc circuit equations can be found either by the Y*> correspond to all combinations of stable cell cqui- 
brute force algorithm or by using some more efficient Kbrium points, then S 0 can have at most 2 m members, 
algorithms known in the art However, even for this 20 henceforth called the set of virtual equilibrium output 
very simple example, there are 32 unknown variables vectors. We use the adjective "virtual" to emphasize 
and 32 equations (16 linear equations and 16 piecewise- th at some members of So may not be valid equilibrium 
linear equations). It is time consuming to find all of the points of the overall circuit. This is because some of the 
equilibrium points of a cellular neural network by using arbitrary combinations of V y u= ± 1 used in calculating 
the algorithms mentioned above because of the large 25 the circuit equilibrium states may not correspond to 
size of its circuit equations. (Note that if the nonlinearity actual solutions obtained by solving the complete cou- 
of the circuit is not piecewise-linear, there is no general pled systems of algebraic equations (2) with the left- 
method to find all of its equilibrium points.) hand side of equation (la) set equal to zero. 

To simplify the problem, the various features of cellu- _ ~ n u . , c „ . . c . ~ .... . _ . 
tor neural I networks can take advantage of the analysis. 30 De,5mt,0,, 3: Stable Eq^nun. Pent 

As mentioned before, every cell in a cellular neural A stable system equilibrium point of a cellular neural 
network has the same connections as its neighbors. network with A(ij;k,l)> l/K x is any equilibrium state 
Therefore, each cell's circuit equation is the same as vector, V* whose state variable components, V*^ (Lc, 
that of the other cells in the same circuit. (Without loss all capacitor voltages) consist of stable cell equilibrium 
of generality, boundary effects can be ignored.) Hence, states. 

the global properties of a cellular neural network can be It is important in the following analysis to remember 

understood by studying the local properties of a single that whereas all stable system equilibrium points are 

cell. This approach is extremely useful for the analysis made up of stable cell equilibrium states, the converse is 

and design of cellular neural networks. not true, as some combinations of stable cell equilibrium 

Before analyzing this example, it is helpful to recall points may not correspond to actual equilibrium points 

the following definitions: of the overall circuit If we denote the corresponding 

- . . t _ „ _ .... . _ set S 0 * of observable equilibrium output vectors, then 

Definition 1: Cell Equilibrium State we flave s<» [s^S 

A cell equilibrium state V*^of a cell circuit C(i j) in Now, let us compute the stable cell equilibrium states 
a cellular neural network with dc input voltages V v */is of an inner cell of the preceding circuit example. Con- 
any value of the state variable V^y which satisfies sidering the condition in the above definitions, the sta- 
ble cell equilibrium states can be obtained by solving 
(39a) equation (38a) for with V y ij taking on either the 
W £&L\ =o „ value +1 or —I: 



(b) vykl = ±1 for all neighbor cells C WWAffr (39b) 



K x ^ J8 n[K^_ I ]+2ig7i[^+ JK n[^ +I l (41a) 
| Fxi/t £M£/3Afr l£j£N. (41b) 

It follows from Definition 1 that the set of all cell 
equilibrium states of a cell circuit CQ4) can be found by 55 Substituting i and \ in the above equations 
equating the right-hand side of equation (la) to zero and by ±1, and considering that sgn [V^] =sgn[V xff ] from 
by setting V,*/ to be any combination of ± 1, and then equation (38b), we obtain the following four cases: 
solving for V^y. Note that since the resulting equation is 
decoupled from the rest, it can be trivially solved. On 

the other hand, many of these solutions could be extra- 60 < a > Fw M-i = -i andv^+i = -l, we have 
neous since when the entire coupled system equation (1) ^ = _2 + lign^^], and hence txg = -4. 

is solved for its equilibrium points, some combinations 

of Vyij= ± 1 may not satisfy these equations. {b) For ^ „ + ! and p^+i = -1. we have 

Observe also that it is possible for a cell equilibrium 
state to have an absolute value less than one, in which 65 >^ = 2^[^, and hence - -2, or 2. 

case, even if this is a valid solution, it would not be 

observable (i.e., unstable)in view of equations (28M30), (c) For *W-» = -lwd w+ i = +l.wehave 

assuming that A(i j;kj)> 1/R*. We will usually be inter- *xy - 2sg«[^> and hence ^ - -2, or z 
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-continued output images by different cellular neural networks for 

different image processing or pattern recognition pur- 

(d) For »yij-\ - 1 and k^+i = 1, we have poses. 

- ^ . r , . . A The indexes of nodes are coded as follows: The first 

>*-2 + 2v4**«ifa«*f-i 5 number from the left specifies the type of the nodes in a 

It follows from the above analysis that the stable cell !J^ # ? 2? ™ lte *^ 2 /or the output volt- 

equilibrium states of any inner cell circuit For our pres- f ge noA *£ e ™ mb *{ fr ° m ^ 

ent example are -4, -2, 2, and 4 layer number; 41,6 i}md Md fourth numbers «P«ry 

It is obvious that the stable cell equflibrium state of 10 rowS; ™ d ^ ^ f" 1 11,6 bl- 
each ceU depends on the stable ceU equilibrium states of ^P 1 ^ v 0^102) means V x n, and 
its neighbor cells. Of course, if the input of the cellular v (2 10104) means and so on. 
neural network is not zero, then the stable cell equilib- Multiplexing is one of the various ways to input data 
rium states of a cell circuit will also depend on the input, to cxtract data from the individual cells of a CNN. 
V«. Therefore, the stable system equilibrium points of a nG - 13 shows a simplified 4X4 CNN that consists of 
cellular neural network by Definition 3 depend on the individual CNN cells 140-x where x ranges from 1 to 16. 
initial conditions, the inputs, and the dynamic rule of the Each CNN cell has connected to it a corresponding 
circuit Any stable system equilibrium point can be individual data switch 142-x where x is the same as for 
considered as a two-dimensional pattern. For a given the individual cells. Each of switches 142-x are con- 
cellular neural network, like the one in this example, _ ft trolled in groups of four by control lines 146-y where y 
there are only a finite number of discrete patterns. The ranges from 1 to 4 to effectively correspond with the 
continuous image space, [— l,l] MxAr , therefore, is trans- columns of CNN cells in the array. Each of switches 
formed into a finite number of discrete patterns by a 142-x are also connected in groups of four to data lines 
cellular neural network. The properties of the patterns 144-z where z ranges from 1 to 4 to effectively corre- 
can be determined by those of the stable cell equilibrium spond with the rows of CNN cells in the array. In this 
states. Consequently, the output voltage V y must pos- configuration, an analog input voltage is applied to each 
sess some structural features for a specific cellular neu- of the data lines 144-2 and one of the control lines 146-y 
ral network. For example, if V^_i= 1, and V^-j- 1 is selected to apply those signals to the CNN cells in a 
as prescribed by the above case (d), then V^= 1 is the particular row of CNN cells. Thus, each column is 
only choice. loaded sequentially with each cell in that column loaded 

As noted above, the image transform by a cellular *> m parallel. This data transfer technique could be per- 

neural network is a dynamical transform. So, it is impor- formed in any order that one may desire, 

taut to consider the transient response of a cellular neu- By ^ owkig & e ^^1 ^ t0 ^ve ^ levekf 

ral network. The transient response of a ceUular neural ^ ^ ^ 144 _ 2 ^ m u ^ ^ mput 

network is simply the trajectory from the initial state to ou ^ ut ^ potabmy h to ^ ^ 

t^steady state of the system It is well known from the « ^consist of two wirHnichwould to d£ 

w ^ states for the two control lines. OneVtate could be 

equilibnum pointas defined in Definition 3, of a eel u- ^ Qn ^ — third A 

lar neural network is a limit point of a set of trajectories * \ . ^^-i, ««« » UUWIU,C4 ' 1 * **™«™ 

of the corresponding differential equations (1). Such an ncct T^rZ^u W te . ncc ff ar y to 

attracting limit poift has a tataKSffi 7%£> « ^ *T " * ^ 

consists of the imion of all trajectories tending to this t0 *T; * *? discreet opUon were not 

limit point. The state space of the system is partitioned P™*** * ^ m * X ° 

into a set of basins centered at the stable system equilib- t0 a * ea S ^ or rt Jf*" 1 ? 8"*^ P° wcr 

rium points. Then the map, F, as defined in equation f 0 ^^ Multl P 1 ^8 ^° mterfacing to a 

(36), can be characterized by the basins and, hence, by « ^N? ™* fewer terminals than would be 

the transient response. Since it is extremely difficult if ^J""* rf « "J Y*?™* data ^ were provided for 

not impossible to analyze the transient behavior of such ^ of mc 0,14 m y- Additionally, multiplexing 

a complex nonlinear dynamical system, a computer * especially useful when the data to be input into the 

simulation method has been used to study the transient CNN » supplied by a computer via a high-speed 

behaviors of ceUular neural networks in the following 50 D/A converter or from a remote sensor, 

examples. Referring next to FIG. 14 there is shown a simplified 

Summarizing the above observations from this exam- 4x4 CNN **** consists of individual CNN cells 148-x 

pie, where x ranges from 1 to 16. Each CNN cell has con- 

(a) Any input image represented in analog form can nected to it a corresponding individual optical sensor 
be mapped into a specific output image with binary 55 150-x where x is the same as for the individual CNN 
values by a cellular neural network, cells. Each of the optical sensors can be used to opti- 

(b) For a given cellular neural network, the output cally detect a portion of an image that represents the 
images are imbued with some spatial structures result- input data to the corresponding CNN cell. In this con- 
ing from the dynamic rule of the circuit for »manre r it figuration, the data or image to be processed would be 
is impossible to have a row like [1,-1,1,-1], which is a 60 projected directly onto the CNN chip through a trans- 
rejected partem, in the output image of the cellular parent window in the chip package. Each cell of the 
neural network in the above example. Hence, an appro- CNN would then receive its input signal in parallel with 
priately chosen dynamic rule could imbue a cellular each of die others at a very high speed. This configura- 
neural network with the ability to recognize and extract ' tion could be used where the CNN is being used for 
some special patterns from input images. 65 pattern recognition with cell of the CNN dedi- 

(c) Different input images may be mapped into the cated to process a single pixel of the input image. Thus, 
same output image if they have the same patterns, and the template of the CNN would be designed to identify 
the same input image may be mapped into different a particular image, and if all of the pixels (or a selected 
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percentage of the pixels) of an applied image agree with 
the template element in each cell of the CNN, the ap- 
plied image is identified as that for which the CNN 
array was programmed to recognize. This configura- 
tion therefore allows a CNN processor to act on the 
input image as it comes directly from a real-world im- 
age. 

The output signals from the CNN array can be han- 
dled in many different ways including direct output 
from each cell or through multiplexing. In addition, 
either of those techniques could be implemented elec- 
tronically or optically. An optical input/output tech- 
nique would lend itself to extremely high speed applica- 
tions, particularly in an image identification or enhance- 
ment application. In this configuration, the optical sen- 
sor/transmitter would produce photoelectrons when 
exposed to light in the input mode, and radiate a light 
having an intensity that is proportional to the output 
voltage of the CNN cell in the output mode. 

To those skilled in the art to which this invention 20 
relates, many changes in construction and widely differ- 
ing embodiments and applications of the invention will 
suggest themselves without departing from the spirit 
and scope of the invention. The disclosures and the 
descriptions herein are purely illustrative and are not 
intended to be in any sense limiting. 

What is claimed is: 

1. A cellular neural network comprising: 

a plurality of identical analog cells arranged in a mul- 3Q 
ti-dimensional lattice array of said plurality of cells, 
the cells closest to any single cell in each of the 
multiple dimensions being neighbor cells that form 
a first layer of neighbor cells that are intercon- 
nected with said any single cell in the same way 33 
depending on their relative position as a neighbor 
cell to said any single cell, all of the cells separated 
from any single cell by one other cell in any of the 
multiple dimensions form a second layer of neigh- 
bor cells that are interconnected with said any 40 
single cell in the same way depending on their 
relative position as a neighbor cell to said any sin- 
gle cell, and the layers of neighbor cells expanding 
concentrically from said second layer of neighbor 
cells in the multiple dimensions around said any 45 
single cell forming additional layers of neighbor 
cells to the full extent of said lattice array that are 
interconnected with said any single cell in the same 
way depending on their relative position as a neigh- 
bor cell to said any single cell, 

each of said cells interacting non-linearly and contin- 
uously in time with selected layers of said neighbor 
cells and at least said first layer of each of said cell's 
nearest neighbor cells with independently selected 
parameters for each of cell of said layers depending 55 
on the relative position of each cell in said layer to 
said any single cell, 

wherein each cell includes: 

non-linear feed-forward means coupled to each cell in 
the selected layers of said each cell's neighbor cells 60 
for influencing the state of each cell in said selected 
layers of neighbor cells, 

non-linear feedback means coupled to each cell in the 
selected layers of each cell's neighbor cells for 
influencing the state of each cell in said selected 65 
layers of said neighbor cells, and 

non-linear self feedback means for influencing the 
state of itself. 



50 



2. The cellular neural network of claim 1 wherein 
said array of cells is an n-dimensional array wherein n is 
an integer number of at least 2. 

3. The cellular neural network of claim 2 wherein n 
equals 2 with the array of cells having i rows and j 
columns, i and j being independent integers of at least 2. 

4. The cellular neural network of claim 1 wherein 
there are m levels of neighbors wherein m is an integer 
number of at least 1. 

5. The cellular neural network of claim 1 wherein 
said feed-forward means of each cell is controlled by an 
independent signal source. 

6. The cellular neural network of claim 1 wherein 
said feedback and self feedback means of each cell are 
each controlled by the state of its cell. 

7. The cellular neural network of claim 1 wherein: 
said feed forward means of each of said any single cell 

includes multiple output voltage controlled current 
source means for generating fixed feed forward 
signals to the cells in each layer of neighbor cells 
depending on relative position of the neighbor cell 
to said any single cell, and 
said feedback and self feedback means of each of said 
any single cell includes multiple output voltage 
controlled current source means for generating 
fixed feedback signals to the cells in each layer of 
neighbor cells depending on relative position of the 
neighbor cell to said any single cell and the same 
self feedback signal to said any single cell. 

8. The cellular neural network of claim 1 wherein 
each cell interacts with a selected number of contiguous 
layers of the cell's nearest neighbor cells. 

9. A pattern recognition system comprising: 

a plurality of cellular neural networks each function- 
ing concurrently with each other to substantially 
simultaneously identify a different particular por- 
tion of the pattern of interest and to signal if that 
portion of the pattern was detected from an input 
signal that is simultaneously applied to each of said 
plurality of cellular neural networks, and 

decision means coupled to receive said signal from 
each of said plurality of cellular neural networks 
for determining if each element of the pattern of 
interest is present, 

each of said cellular neural networks includes: 

a plurality of identical analog cells arranged in a mul- 
ti-dimensional lattice array of said plurality of cells, 
the cells closest to any single cell in each of the 
multiple dimensions being neighbor cells that form 
a first layer of neighbor cells that are intercon- 
nected with said any single cell in the same way 
depending on their relative position as a neighbor 
cell to said any single cell, all of the cells separated 
from any single cell by one other cell in any of the 
multiple dimensions form a second layer of neigh- 
bor cells that are interconnected with said any 
single cell in the same way depending on their 
relative position as a neighbor cell to said any sin- 
gle cell, and the layers of neighbor cells expanding 
concentrically from said second layer of neighbor 
cells in the multiple dimensions around said any 
single cell forming additional layers of neighbor 
cells to the full extent of said lattice array that are 
interconnected with said any single cell in the same 
way depending on their relative position as a neigh- 
bor cell to said any single cell, 

each of said cells interacting non-linearly and contin- 
uously in time with selected layers of said neighbor 
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cells and at least said first layer of each of said cell's * neighbor cells depending on relative position of the 
nearest neighbor cells with independently selected neighbor cell to said any single cell and the same 

parameters for each of cell of said layers depending self feedback signal to said any single cell, 

on the relative position of each cell in said layer to 14. The pattern recognition system of claim 9 
said any single cell, 3 wherein each cell within each of the cellular neural 

wherein each cell includes: networks is disposed to receive a corresponding single 

non-linear feed-forward means coupled to each cell in pixel of a display that is the subject of the desired pat- 
the selected layers of said each cell's neighbor cells tern recognition. 

of influencing the state of each cell in said selected 15. The pattern recognition system of claim 14 fur- 
layers of neighbor cells, 10 fo a includes a plurality of optical detector means with 

non-linear feedback means coupled to each cell in the ^ gucn means connected to each cell of each cellular 
selected layers of each cell's neighbor ceDs for neural network for being optically exposed to the corre- 
influencing the state of each cell in said selected spending pixel of the subject pattern recognition display 
layers of said neighbor cells, and for oftnat pixd mto the cell. 

non-linear self feedback means for influencing the 15 16 ^ recognition system of claim 9 further 

-* Sta i^ of itsclf ' , , . including multiplex means coupled to each cell in each 

10. The pattern recognition system of claim 9 ^ for of thc m signalinto 

wherem each cell mteracts with a seated number of consecutively a row or column at a time to 

contiguous layers of the cell's nearest neighbor cells. . . . * r #( ^- lc rM1 Kv 

11. The pattern recognition system of claim 9 20 ™niini« the number of external terminals required by 

wherein said feed-forward means of each cell is con- "ie system. - . . . 

trolled by an independent signal source. . ^J** pa ?f? nco « DlUon ^ ° f 

12. lie r^tte^rccognkion system of claim 9 including mutoplex means coupled to each cell in each 
wherem said feedback and self feedback means of each **** "*» ork for ^traction of the outpu 
ceU are each controlled by the state of its cell. 25 from ^ consecutively a row or column at 

13. The pattern recognition system of claim 9 a time to ininimize the number of external terminals 
wherein: required by the system. 

said feed forward means of each of said any single cell 1* The pattern recognition system of claim 9 further 

includes multiple output voltage controlled current including multiplex means coupled to each cell in each 

source means for generating fixed feed forward 30 cellular neural network for entry of the input signal into 

signals to the cells in each layer of neighbor cells the cells consecutively a row or column at a time prior 

depending on relative position of the neighbor cell to the processing of the information, for extraction of 

to said any single cell, and the output signal from the cells consecutively a row or 

said feedback and self feedback means of each of said column at a time after the processing of the signal, and 
any single cell includes multiple output voltage 35 for minimizing of the number of external terminals re- 
controlled current source means for generating quired by the system, 
fixed feedback signals to the cells in each layer of * * * * * 
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